Russian Church
If you're spending a few days in Gyumri, make your way over to this church, which is a bit different from the norm.
Town Square
Huge square in the middle of town, with a church, a cathedral, a massive government building, fountains, restaurants, and all adjacent to the old town with great architecture and some small museums.
National Arch. Museum
This museum is nearer the highway to Yerevan, and has beautiful traditional architecture, interesting rooms, and a cool dining room where you can order a meal ahead of time.
Black Fortress
The big, round, black fortress on the top of the hill overlooking Gyumri is a good spot to take in the view. It once protected the Russian Empire from the Ottoman Empire. Nearby is the huge "Mother Armenia" statue.